Abstract

Quantitative meta-analysis is a very useful, yet underutilized, technique for synthesizing research findings in higher education. Meta-analytic inquiry can be more challenging in higher education than in other fields of study as a result of (a) concerns about the use of regression coefficients as a metric for comparing the magnitude of effects across studies, and (b) the non-independence of observations that occurs when a single study contains multiple effect sizes. This methodological note discusses these two important issues and provides concrete suggestions for addressing them. First, meta-analysis scholars have concluded that standardized regression coefficients, which are often provided in higher education manuscripts, constitute an appropriate metric of effect size. Second, hierarchical linear modeling (HLM) analyses provide an effective method for conducting meta-analytic research while accounting for the non-independence of observations, and HLM is generally superior to other proposed methods that attempt to remedy this same problem. A discussion of how to implement these techniques appropriately is provided.
Hide

Abstract

Overviews that combine single effect estimates from published studies generally use a summary statistic approach where the effect of interest is first estimated within each study and then averaged across studies in an appropriately weighted manner. Combining multiple regression coefficients from publications is more problematic, particularly when there are differences in study design and inconsistent reporting of effect sizes and standard errors. This paper describes the use of a hierarchical model in such circumstances. Its use is illustrated in a meta-analysis of the metabolic wa:rd studies that have investigated the effect of changes in intake of various dietary lipids on blood cholesterol. These studies all reported average blood cholesterol for groups of individuals who were studied on one or more diets. Thirty-one studies had randomized cross-over designs, 12 had matched parallel group designs, 12 had non-randomized Latin square designs and 16 had other uncontrolled designs. The hierarchical model allowed the different types of comparison (within-group between-diet, between matched group) that were made in the various studies to each contribute to the overall estimates in an appropriately weighted manner by distinguishing between-study variation, within-study between-matched-group variation and within-group between-diet variation. The hierarchical models do not require consistent specification of effect sizes and standard errors and hence have particular utility in combining results from published studies where the relationships between a dependent variable and two or more predictors have been investigated using heterogeneous methods of analysis. Copyright (C) 1999 John Wiley & Sons, Ltd.
Hide

Abstract

Regression methods are widely used by researchers in many fields, yet methods for synthesizing regression results are scarce. This study proposes using a factored likelihood method, originally developed to handle missing data, to appropriately synthesize regression models involving different predictors. This method uses the correlations reported in the regression studies to calculate synthesized standardized slopes. It uses available correlations to estimate missing ones through a series of regressions, allowing us to synthesize correlations among variables as if each included study contained all the same variables. Great accuracy and stability of this method under fixed-effects models were found through Monte Carlo simulation. An example was provided to demonstrate the steps for calculating the synthesized slopes through sweep operators. By rearranging the predictors in the included regression models or omitting a relatively small number of correlations from those models, we can easily apply the factored likelihood method to many situations involving synthesis of linear models. Limitations and other possible methods for synthesizing more complicated models are discussed. Copyright © 2012 John Wiley & Sons, Ltd.
Hide

Abstract

The partial correlation and the semi-partial correlation can be seen as measures of partial effect sizes for the correlational family. Thus, both indices have been used in the meta-analysis literature to represent the relationship between an outcome and a predictor of interest, controlling for the effect of other variables in the model. This article evaluates the accuracy of synthesizing these two indices under different situations. Both partial correlation and the semi-partial correlation appear to behave as expected with respect to bias and root mean squared error (RMSE). However, the partial correlation seems to outperform the semi-partial correlation regarding Type I error of the homogeneity test (Q statistic). Although further investigation is needed to fully understand the impact of meta-analyzing partial effect sizes, the current study demonstrates the accuracy of both indices.
Hide

Abstract

Combining multiple regression estimates with meta-analysis has continued to be a difficult task. A variety of methods have been proposed and used to combine multiple regression slope estimates with meta-analysis, however, most of these methods have serious methodological and practical limitations. The purpose of this study was to explore the use of robust variance estimation for combining commonly specified multiple regression models and for combining sample-dependent focal slope estimates from diversely specified models. A series of Monte-Carlo simulations were conducted to investigate the performance of a robust variance estimator for each of these approaches. Key meta-analytic parameters were varied throughout the process. Also, two small scale, examples were conducted to illustrate the use of the robust variance estimator in each of these two approaches. In general, the robust variance estimator performed well. Robust confidence interval parameter recovery was close to the specified 95% under almost all conditions. Only when there were a larger number of slope estimates and a small number of study samples did the robust standard errors noticeably lose efficiency. Combining sample-dependent focal slope estimates provides biased point estimates, however, the results of this paper suggest that the robust standard errors are still accurate.
Hide

Abstract

When conducting a meta-analysis, it is common to find many collected studies that report regression analyses, because multiple regression analysis is widely used in many fields. Meta-analysis uses effect sizes drawn from individual studies as a means of synthesizing a collection of results. However, indices of effect size from regression analyses have not been studied extensively. Standardized regression coefficients from multiple regression analysis are scale free estimates of the effect of a predictor on a single outcome. Thus these coefficients can be used as effect-size indices for combining studies of the effect of a focal predictor on a target outcome. I begin with a discussion of the statistical properties of standardized regression coefficients when used as measures of effect size in meta-analysis. The main purpose of this dissertation is the presentation of methods for obtaining standardized regression coefficients and their standard errors from reported regression results. An example of this method is demonstrated using selected studies from a published meta-analysis on teacher verbal ability and school outcomes (Aloe & Becker, 2009). Last, a simulation is conducted to examine the effect of multicollinearity (intercorrelation among predictors), as well as the number of predictors on the distributions of the estimated standardized regression slopes and their variance estimates. This is followed by an examination of the empirical distribution of estimated standardized regression slopes and their variances from simulated data for different conditions. The estimated standardized regression slopes have larger variance and get close to zero when predictors are highly correlated via the simulation study.
Hide

Abstract

In this article we focus on three partial effect sizes for the correlation (r) family of effects: the standardized slope (b), the partial correlation (r_p), and the semi-partial correlation (r_sp). These partial effect sizes are useful for meta-analyses in two common situations: when primary studies reporting regression models do not report bivariate correlations, and when it is of specific interest to partial out the effects of other variables. We clarify the use of these three indices in the context of meta-analysis and describe how the indices can be estimated and analyzed. We provide examples of syntheses of these partial effect sizes using a published social work meta-analysis. Finally, we share practical recommendations for meta-analysts wanting to use such indices.
Hide

Abstract

A new effect size representing the predictive power of an independent variable from a multiple regression model is presented. The index, denoted as rsp, is the semipartial correlation of the predictor with the outcome of interest. This effect size can be computed when multiple predictor variables are included in the regression model and represents a partial effect size in the correlation family. The derivations presented in this article provide the effect size and its variance. Standard errors and confidence intervals can be computed for individual rsp values. Also, meta-analysis of the semipartial correlations can proceed in a similar fashion to typical meta-analyses, where weighted analyses can be used to explore heterogeneity and to estimate central tendency and variation in the effects. The authors provide an example from a meta-analysis of studies of the relationship of teacher verbal ability to school outcomes.
Hide

Abstract

Research on methods of meta-analysis (the synthesis of related study results) has dealt with many simple study indices, but less attention has been paid to the issue of summarizing regression slopes. In part this is because of the many complications that arise when real sets of regression models are accumulated. We outline the complexities involved in synthesizing slopes, describe existing methods of analysis and present a multivariate generalized least squares approach to the synthesis of regression slopes.
Hide

Abstract

Regression is one of the most commonly used quantitative methods for exploring the relationship between predictor(s) and the outcome of interest. One of the challenges meta-analysts may face when intending to combine results from regression studies is that the predictors are usually different from study to study, even though the primary researchers may have been studying the same outcome. In the current study, generalized least squares (GLS) and factored likelihoods through the sweep operator (SWP), for combining results were examined for their ability to reduce the problems arising from regression models that contain different predictors in the meta-analysis. Both of the methods utilize the zero-order correlations among the variables in the regression studies. The GLS method treats the correlations from each study as a subset of multivariate outcomes, and combines the results with the consideration of the dependencies of the correlations across studies (Raudenbush, Becker, & Kalaian, 1988; Becker, 1992). The SWP method in this study applies the concept of missing data to the regression models that contained different predictors included in the synthesis. An empirical study was conducted by creating a set of regression studies using a subset of NELS:88 dataset. The correlations among the created studies were combined. A final regression model with standardized slopes was calculated for each of the predictors using each of the two methods. The results from this empirical study showed that SWP produced less biased estimates of slopes in most situations. The precision of the results from those two methods could be impacted by the features of studies included in the meta-analysis. Therefore, a simulation was conducted to investigate the impacts of missing-data patterns, intercorrelations among the predictors and the outcome, and the sample size. The results indicated that the difference between the two methods was not large. SWP consistently performed slightly better at estimating the slope of the predictor that was fully observed in all studies in the synthesis. Generally, SWP performed well when the sample sizes were equal and small across all studies, and GLS performed better when the sample sizes were equal and large.
Hide

Abstract

A new approach to representing data from multiple regression designs is presented in this dissertation. The index, denoted as r sp , is the semi-partial correlation of the predictor with the outcome of interest. This effect size can be computed when multiple predictor variables are included in the regression model, and represents a partial effect size in the correlation family. The derivations presented in this dissertation provide the partial effect size and its variance. Standard errors and confidence intervals can be computed for individual rsp values. Also, meta-analysis of the semi-partial correlations can proceed in a similar fashion to typical meta-analyses weighted analyses can be used to explore heterogeneity and to estimate central tendency and variation in the effects. A simulation study is presented to study the behavior of this index and its variance.
Hide